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The  Honorable  William  V.  Roth,  Jr. 

The  Honorable  David  Pryor 
United  States  Senate 

This  report,  which  was  prepared  at  your  request,  addresses  the  Department  of  Defense’s 
policies  and  practices  on  the  use  of  low-rate  initial  production  in  relation  to  operational  test  and 
evaluation.  It  also  examines  the  impact  of  the  current  legislative  requirements  in  each  area.  We 
make  recommendations  to  the  Secretary  of  Defense  and  present  matters  for  congressional 
consideration,  both  of  which  are  intended  to  minimize  the  risk  of  prematurely  committing  to 
production  and  later  encoimtering  costly  technical  problems. 

As  agreed  with  your  office,  unless  you  publicly  announce  its  contents  earlier,  we  plan  no  further 
distribution  of  this  report  until  30  days  from  its  issue  date.  At  that  time,  we  will  send  copies  to 
the  Chairmen  and  Ranking  Minority  Members  of  the  House  Committee  on  Government 
Operations,  Senate  Committee  on  Governmental  Affairs,  House  and  Senate  Committees  on 
Armed  Services,  and  House  and  Senate  Committees  on  Appropri^ons;  the  Secretaries  of 
Defense,  the  Army,  the  Air  Force,  and  the  Navy;  and  the  Director  of  the  Office  of  Management 
and  Budget.  We  will  also  make  copies  available  to  others  upon  request. 

This  report  was  prepared  under  the  direction  of  Louis  J.  Rodrigues,  Director  of  Systems 
Development  and  Production  Issues,  who  may  be  reached  on  (202)  5124841  if  you  or  your  staff 
have  any  questions.  Other  major  contributors  are  listed  in  ^pendix  H. 


Henry  L.  Hinton,  Jr. 

Assistant  Comptroller  General 
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Executive  Summary 


Purpose 


The  Department  of  Defense  (dod)  plans  to  spend  over  $79  billion  in  fiscal 
year  1995  for  the  acquisition  of  weapon  systems,  dod  has  acquired  some  of 
the  most  technologically  advanced  and  effective  weapon  systems  in  the 
world.  However,  dod  has  often  been  criticized  for  not  acquiring  the 
systems  in  the  most  efficient  manner. 

Because  of  concern  over  how  dod  initiates  the  production  of  weapon 
systems,  Senators  David  Pryor  and  William  V.  Roth,  Jr.,  requested  that  GAO 
review  dod’s  use  of  a  practice  called  low-rate  initial  production  (lrip)  in 
system  acquisition  programs.  Specifically,  GAO  was  requested  to  determine 
whether  lrip  practices  were  resulting  in  production  of  systems  with 
adequate  performance  and  whether  the  legislation  underlying  lrip  policies 
was  adequate. 


Background 


The  Congress  has  been  concerned  about  the  sometimes  significant 
quantity  of  systems  produced  during  lrip  and  before  system  performance 
is  operationally  demonstrated.  As  a  result,  in  1989,  the  Congress  enacted 
legislation  intended  to  limit  lrip  quantities  for  major  systems.  The  law, 

10  U.S.C.  2400,  provides  that,  except  for  ships  and  satellites,  lrip  is 
production  in  the  minimum  quantity  necessary  to  (1)  provide 
production-configured  or  representative  articles  for  operational  test  and 
evaluation  (ot&e),  (2)  establish  an  initial  production  base  for  the  system, 
and  (3)  permit  an  orderly  increase  in  the  production  rate  for  the  system 
sufficient  to  lead  to  fuU-rate  production  upon  the  successful  completion  of 
OT&E. 

OT&E  is  the  primary  means  of  assessing  weapon  system  performance  in  a 
combat-representative  environment  It  is  defined  as  (1)  the  field  test, 
conducted  under  realistic  conditions,  to  determine  an  item’s  effectiveness 
and  suitability  for  use  in  combat  by  typical  military  users  and  (2)  the 
evaluation  of  the  results  of  such  a  test  If  used  effectively,  ot&e  can  be  a 
key  internal  control  to  ensure  that  decisionmakers  have  objective 
information  available  on  a  weapon  system’s  performance,  thereby 
minimizing  risks  of  procuring  costly  and  ineffective  systems. 

As  weapon  system  programs  move  through  the  phases  of  the  acquisition 
process,  they  are  subject  to  review  at  mcyor  decision  points  called 
milestones.  The  milestone  review  process  is  predicated  on  the  principle 
that  systems  advance  to  higher  acquisition  phases  by  demonstrating  that 
they  have  met  prescribed  technical  specifications  and  performance 
thresholds.  Before  systems  advance  to  the  milestone  that  authorizes 
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full-rate  production,  dod  is  statutorily  required  to  conduct  ot&e  for  all 
mqjor  systems. 


Results  in  Brief 


Current  legislation  and  dod’s  acquisition  policies  permit  LRIP  to  start  before 
any  ot&e  is  conducted.  The  consequences  have  included  procurement  of 
substantial  inventories  of  unsatisfactory  weapons  requiring  costly 
modifications  to  achieve  satisfactory  performance  and,  in  some  cases, 
deployment  of  substandard  systems  to  combat  forces.  Once  started,  lrip 
significantly  limits  the  options  available  to  dod  decisionmakers  and  the 
Congress  when  a  system  is  found  to  be  deficient.  In  GAO’s  view,  the  key 
decision  as  to  whether  to  proceed  with  production  should  be  made  at  the 
start  of  LRIP  because,  in  many  cases,  it  is  also  the  de-facto  fuU-rate 
production  decision.  Therefore,  decisionmakers  need  good,  independent 
information  on  the  system’s  performance  and  suitability  at  that  point.  In 
today’s  national  security  environment,  there  should  be  very  few  cases  in 
which  an  urgent  need  dictates  that  dod  start  lrip  without  a  demonstrated 
level  of  confidence  that  the  system  will  work  as  intended. 


Principal  Findings 


Current  Legislation  and 
DOD  Acquisition  Policies 
Permit  LRIP  to  Begin 
Without  Any  OT&E 


There  are  no  specific  principles  or  guidelines — ^in  legislation  or  dod 
policy — on  when  and  how  programs  should  begin  lrip,  on  the  type  and 
amount  of  testing  to  be  done  before  lrip,  on  how  much  lrip  can  or  should 
be  done,  or  under  what  type  of  circiunstances  lrip  should  be  curtailed  or 
stopped.  As  a  result,  programs  are  often  permitted  to  begin  lrip  with  little 
or  no  scrutiny  and  before  any  OT&E  has  been  conducted.  Although 
programs  are  sometimes  delayed  in  getting  £q)proval  for  fioU-rate 
production,  lrip  is  not  usually  stopped  or  slowed  down  significantly. 


Controls  Needed  Over  Over  the  years,  GAO  has  found  numerous  instances  in  which  production  of 

LJJJP  both  major  and  nonmajor  systems  from  all  of  the  services  was 

optimistically  permitted  to  begin  under  lrip  and  continue  based  on  factors 
other  than  the  systems’  technical  maturity.  Many  of  the  weapon  systems 
that  start  production  prematurely  later  experience  significant 
effectiveness  and/or  suitability  problems.  For  example,  the  reUabitity  of 
the  Air  Force’s  C-17  aircraft  has  shown  to  be  significantly  less  than 
expected,  and  it  cannot  meet  its  minimum  payload  and  range 
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specifications.  M^or  design  changes  are  often  needed  to  correct  the 
problems  and  additional  testing  is  needed  to  verify  that  the  corrective 
action  was  effective.  For  example,  major  design  changes  to  correct 
deficiencies  in  the  Navy’s  T-45A  aircraft  included  a  new  engine  and  new 
wings.  When  problems  are  uncovered  after  production  starts,  costly 
retrofits  are  often  needed  for  any  delivered  units.  Also,  problems  have 
sometimes  not  been  fixed,  and  substandard  systems  have  been  deployed 
to  field  units.  For  example,  although  considered  an  effective  we^on 
system,  the  Army’s  Apache  hehcopter  has  proven  to  be  a  difficult  and 
costly  system  to  support.  Although  the  thrust  of  the  lrip  legislation  is  to 
authorize  only  minimum  necessary  quantities,  the  continuance  of  lrip  on 
an  indefinite  basis  has  resulted  in  mgjor  production  commitments.  For 
example,  while  the  original  plans  called  for  the  production  of  810 
Advanced  Medium  Range  Air-to-Air  Missiles  during  2  years  of  lrip,  over 
4,100  were  eventually  produced  during  its  6  years  of  lrip. 

In  a  1993  report,  the  dod  Inspector  General  stated  that  mgjor  acquisition 
programs  were  entering  lrip  without  meeting  development,  testing,  and 
production  readiness  prerequisites.^  The  Inspector  General  recommended 
that  DOD  (1)  provide  guidance  for  all  programs  on  the  specific  minimum 
required  program  accomplishments  for  entry  into  and  continuation  of  LRIP 
and  (2)  require  that  program-specific  exit  criteria  be  established  for  entry 
into  and  continuation  of  lrip.  To  reduce  the  risk  of  finding  major 
operational  effectiveness  and  suitability  problems  after  production  starts, 
GAO  has  often  recommended  as  much  ot&e  as  possible  before  production 
starts.  GAO  recognizes  that,  in  addition  to  ship  and  satellite  programs,  it 
may  be  very  costly  and  disruptive  to  suspend  certain  unique  programs 
while  OT&E  is  underway.  'These  programs  would  typically  involve  inherent 
fabrication  complexity,  small  procurement  quantities,  high  unit  cost,  and 
long  unit  production  periods.  However,  one  means  to  reduce  technical 
risks  on  these  programs  would  be  to  conduct  ot&e  of  key  subsystems  on 
surrogate  platforms  before  production  starts. 


Recommendations 


GAO  recommends  that  the  Secretary  of  Defense  revise  dod  system 
acquisition  policies  in  the  following  ways: 

Require  that,  before  entry  into  lrip,  programs  (with  the  exception  of  ships, 
satellites,  and  programs  that  involve  inherent  fabrication  complexity,  small 
procurement  quantities,  high  unit  costs,  and  long  unit  production  periods) 


‘Low-Rate  Initial  Production  in  Major  Defense  Acquisition  Programs  (DOD  Inspector  General  Report 
No.  94-014,  Nov.  9,  1993). 
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plan,  buy  prototypes  for,  and  conduct  enough  realistic  testing  for  the 
services’  independent  testing  agencies  and/or  the  Director,  Operational 
Test  and  Evaluation  (dot&e),  to  be  able  to  certify  to  the  decision  authority 
that  (1)  the  system’s  developmental  testing  is  essentially  complete  and  the 
basic  results  have  been  validated  in  an  operational  environment;  (2)  the 
system  has  clearly  shown  that  it  can  meet  the  key  parameters  among  its 
minimum  acceptable  performance  requirements;^  (3)  the  system  has 
clearly  demonstrated  the  potential  to  fully  meet  all  of  its  minimum 
acceptable  requirements  for  performance  and  suitability  without  mqjor  or 
costly  design  changes;  and  (4)  the  system  should  be  able  to  readily 
complete  its  remaining  ot&e  in  time  to  support  the  planned  fiill-rate 
production  decision.^ 

•  Require  that  those  programs  excluded  from  the  requirement  to  test 
prototypes  instead  test  aU  key  subsystems  in  an  operational  environment 
before  entry  into  lrip. 

•  Adopt  the  recommendations  made  by  the  dod  Inspector  General  regarding 
controls  over  the  start  and  continuation  of  lrip. 

GAO  also  recommends  that  the  Secretary  of  Defense  work  with  the  service 
secretaries  to  ensure  that  these  policies  are  implemented  for  the 
acquisition  of  both  mqjor  and  nonmajor  programs. 


Matters  for 

Congressional 

Consideration 


GAO  recommends  that  the  Congress  legislatively  mandate  (1)  that  certain 
OT&E  requirements  be  met  before  lrip  may  start  and  (2)  specific  limits  on 
the  number  of  units  allowed  to  be  produced  during  lrip.  Specifically,  the 
Congress  may  wish  to  require  that  all  defense  acquisition  programs  (major 
and  nonmajor)  conduct  enough  realistic  testing  on  the  entire  system  or 
key  subsystems  to  ensure  that  key  performance  parameters  are  met  before 
lrip  is  permitted  to  start. 


Agency  Comments 
and  GAO  Evaluation 


In  commenting  on  a  draft  of  this  report,  dod  stated  that  while  much  of  the 
information  reported  by  gao  is  factual,  some  of  the  implications  to  be 
drawn  from  the  information  are  misleading,  dod  questions  the  magnitude 
of  the  problem  and  believes  that  the  current  legislation  provides  sufficient 
guidance  to  limit  the  production  of  lrip  units  without  potentially  causing 


^Minimum  acceptable  requirements  are  the  values  for  the  particular  parameters  that  are  required  to 
provide  a  system  capability  that  will  satisfy  the  validated  mission  need.  These  requirements  are  also 
known  as  thresholds  and  are  established  in  the  Operational  Requirements  Document  at  each 
milestone.  Key  parameters  are  those  for  which  the  decision  authority  would  require  a  reevaluation  of 
alternative  concepts  or  design  approaches  if  the  thresholds  are  not  met. 

®This  certification  should  be  made  by  DOT&E  for  all  m^or  defense  acquisition  programs  and  by  the 
services’  independent  testing  agencies  for  all  other  systems. 
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costly  and  disruptive  gaps  in  production.  DOD  opposes  the  mandating  of 
OT&E  requirements  that  must  be  met  before  the  start  of  production  and 
fixed  hmits  on  the  number  of  units  to  be  produced  in  trap  because  they 
could  cause  production  stretchouts  and  production  stoppages. 

On  numerous  occasions,  GAO  has  reported  on  the  problems  with  premature 
commitments  to  production,  the  inherent  risks  in  highly  concurrent 
acquisition  strategies,  and  the  benefits  of  early  operational  testing. 
However,  dod  continues  to  downplay  or  deny  the  value  of  testing  as  a 
means  of  ensuring  that  weapons  will  work  as  expected.  Such  a  position 
contradicts  the  commonsense  notion  that  quantities  of  products  should  be 
bought  only  after  a  clear  demonstration  that  the  product  actually  works.  In 
this  report,  gao  is  suggesting  additional  legislation  because  dod,  on  its 
own,  has  been  unwilling  or  unable  to  appropriately  control  the  start  and 
continuation  of  production,  despite  the  clear  evidence  that  such  control  is 
needed.  The  objective  of  gao’s  recommendations  is  to  avoid  the  premature 
commitment  to  production  and  thereby  avoid  fielding  systems  that  do  not 
meet  requirements  and  need  costly  and  time-consuming  retrofits. 
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Chapter  1 _ 

Introduction 


In  fiscal  year  1995,  the  Department  of  Defense  (dod)  plans  to  spend  over 
$79  billion  for  research,  development,  test,  evaluation,  and  production  of 
weapon  systems.  While  dod  has  acquired  some  of  the  most  technologically 
advanced  and  effective  weapon  systems,  dod  has  often  been  criticized  for 
not  acquiring  the  systems  in  the  most  efficient  manner. 


System  Acquisition 
Process  and  DOD’s 
Policy 


As  wesqDon  system  programs  progress  through  the  phases  of  the 
acquisition  process,  they  are  subject  to  review  at  mjqor  decision  points 
called  milestones.  The  milestone  review  process  is  predicated  on  the 
principle  that  systems  advance  to  higher  acquisition  phases  by 
demonstrating  that  they  meet  prescribed  technical  specifications  and 
performance  thresholds.  Figiure  1.1  illustrates  the  dod’s  weapon  system 
acquisition  process. 
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Figure  1.1:  DOD’s  Weapon  System  Acquisition  Process 


At  milestone  0,  a  determination  is  made  about  whether  an  identified 
mission  need  warrants  a  study  of  alternative  concepts  to  satisfy  the  need. 
If  warranted,  the  program  is  approved  to  begin  the  concept  exploration 
and  definition  phase.  At  milestone  I,  a  determination  is  made  about 
whether  a  new  acquisition  program  is  warranted.  If  warranted,  initial  cost, 
schedule,  and  performance  goals  are  estabhshed  for  the  program,  and 
authorization  is  given  to  start  the  demonstration  and  validation  phase.  At 
milestone  II,  a  determination  is  made  about  whether  continuation  of 
development,  testing,  and  preparation  for  production  is  warranted.  If 
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warranted,  authorization  is  given  to  start  the  engineering  and 
manufacturing  development  phase.  Also,  approval  of  this  phase  will  often 
involve  a  commitment  to  low-rate  initial  production  (lrip).  At 
milestone  III,  a  determination  is  made  about  whether  the  program 
warrants  a  commitment  to  build,  deploy,  and  support  the  system. 

DOD  acquisition  pohcy  states  that  program  risks  shall  be  assessed  at  each 
milestone  decision  point  before  ^proval  is  granted  for  the  next  phase. 

The  policy  adds  that  test  and  evaluation  shall  be  used  to  determine  system 
maturity  and  identify  areas  of  technical  risk.  Operational  test  and 
evaluation  (ot&e)  is  a  key  internal  control  to  ensure  that  decisionmakers 
have  objective  information  available  on  a  weapon  system’s  performance, 
to  minimize  risks  of  procuring  costly  and  ineffective  systems,  ot&e  has 
been  defined  as  (1)  the  field  test,  imder  realistic  conditions,  of  any  item  of 
(or  key  component  of)  we^ons,  equipment,  or  munitions  for  the  purpose 
of  determining  its  effectiveness  and  suitability  for  use  in  combat  by  typical 
military  users  and  (2)  the  evaluation  of  the  results  of  such  a  test. 


Legislative 

Requirements  on  LRIP 
and  OT&E  of  Weapon 
Systems 


Over  a  period  of  many  years,  the  Congress  has  been  concerned  about  the 
performance  of  weapon  systems  being  acquired  by  DOD.  As  early  as  1972, 
the  Congress  required  DOD  to  provide  it  with  information  on  the  OT&E 
results  of  mcyor  weapon  systems  before  committing  them  to  production. 
However,  the  Congress  continued  to  receive  reports  from  the  DOD 
Inspector  General  (dod-ig),  us,  and  others  that  (1)  we^on  systems  were 
not  being  adequately  tested  before  beginning  production,  (2)  fielded 
systems  were  faihng  to  meet  their  performance  requirements,  and  (3)  OT&E 
being  conducted  on  weapon  systems  was  of  poor  quality. 


In  the  late  1970s  and  early  1980s,  the  Congress  enacted  a  series  of  laws  to 
ensure  that  U.S.  military  personnel  receive  the  best  weapon  systems 
possible  and  that  the  U.S.  government  receives  best  value  for  the  defense 
procurement  dollar.  Among  other  things,  these  laws 


•  specified  that  independent  ot&e  be  conducted; 

•  estabhshed  the  Office  of  the  Director,  Operational  Test  and  Evaluation 
(dot&e),  and  assigned  it  specific  oversight  duties  and  responsibilities; 

•  specified  that  ot&e  of  a  major  defense  acquisition  program  may  not  be 
conducted  until  dot&e  approves  the  adequacy  of  the  plans  for  that  ot&e;* 


*A  “major  defense  acquisition  program”  is  defined  as  a  system  whose  research  and  development  cost  is 
expected  to  exceed  $300  naillion  in  fiscal  year  1990  dollars  or  whose  procurement  cost  is  expected  to 
exceed  $1.8  billion  in  fiscal  year  1990  dollars. 
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•  required  that  a  mjyor  system  may  not  proceed  beyond  lrip  until  its  initial 
OT&E  is  completed;^  and 

•  required  that  dot&e  analyze  the  results  of  OT&E  conducted  for  each  mqjor 
defense  acquisition  program  and,  prior  to  a  final  decision  to  proceed 
beyond  lrip,  report  on  the  adequacy  of  the  testing  and  whether  the  results 
confirm  that  the  items  tested  are  operationally  effective^  and  suitable'*  for 
combat. 

In  the  late  1980s,  the  Congress  found  that  dod  was  acquiring  a  large 
portion  of  the  total  program  quantities,  using  the  lrip  concept,  without 
successfully  completing  OT&E.  In  the  National  Defense  Authorization  Act 
for  Fiscal  Years  1990  and  1991  (P.L.  101-189),  the  Congress  addressed  this 
situation  by  including  a  definition  of  lrip  and  a  requirement  that  the 
determination  of  the  lrip  quantities  to  be  procured  be  made  when  a 
decision  is  made  to  enter  engineering  and  manufacturing  development. 
According  to  the  act,  lrip  was  defined  as  the  minimum  quantity  needed  to 
(a)  provide  production-representative  articles  for  ot&e,  (b)  establish  an 
initial  production  base,  and  (c)  permit  orderly  ramp-up  to  fiill-rate 
production  upon  completion  of  ot&e. 

In  the  conference  report  for  the  act,  the  conferees  indicated  that  they  did 
not  condone  the  continuous  reapproval  of  lrip  quantities  that  eventually 
total  a  significant  percentage  of  the  total  planned  procurement.  Also,  the 
conferees  granted  an  exception  to  the  lrip  legislation  for  ship  and  satellite 
programs  because  of  their  inherent  production  complexity,  small  number, 
high  unit  cost,  and  long  unit  production  periods.  However,  they  directed 
the  Secretary  of  Defense  to  develop  regulations  that  capture  the  spirit  of 
the  LRIP  legislation  as  it  applies  to  these  programs.  This  special 
consideration  for  ships  and  satellites  carries  with  it  additional  reporting 
requirements  to  improve  the  oversight  of  these  programs. 

Finally,  in  the  National  Defense  Authorization  Act  for  Fiscal  Year  1994,  the 
Congress  required  that  the  Secretary  of  Defense  ensure  that  jqipropriate. 


^According  to  DOD  regulation,  a  “major  system”  is  defined  as  a  system  whose  research,  development, 
test,  and  evaluation  cost  is  estimated  to  exceed  $115  million  in  fiscal  year  1990  dollars  or  whose 
procurement  cost  is  estimated  to  exceed  $640  million  in  fiscal  year  1990  dollars. 

®DOD  defines  “operational  effectiveness"  as  the  overall  degree  of  mission  accomplishment  of  a  system 
when  used  by  representative  personnel  in  the  environment  planned  or  expected  for  operational 
employment  of  the  system  considering  organization,  doctrine,  tactics,  survivability,  vulnerability,  and 
threat. 

■•dod  defines  “operational  suitability"  as  the  degree  to  which  a  system  can  be  placed  satisfactorily  in 
field  use  with  consideration  given  to  such  factors  as  availability,  compatibility,  transportability, 
interoperability,  reliability,  wartime  usage  rates,  maintainability,  safety,  and  supportability. 


Page  13 


GAO/NSIAD-95-18  Weapons  Acquisition 


Chapter  1 
Introduction 


rigorous,  and  structured  testing  be  completed  prior  to  lrip  of  any 
electronic  combat  or  command,  control,  and  communications 
countermeasure  system. 


.  Qr'nnp  Senators  David  Pryor  and  William  V.  Roth,  Jr,  requested  that  we  review 

Upj6CtlV6S,  oC0p6,  dod’s  use  of  lrip  in  the  acquisition  of  mqjor  defense  programs.  Specifically, 

and  M6thodolOgy  the  senators  asked  that  we  determine  whether 

•  LRIP  poUcies  were  resulting  in  the  production  of  systems  with  adequate 
performance  capabilities  and 

•  the  legislation  underlying  the  lrip  poUcies  was  adequate. 

We  analyzed  the  legislation  and  dod  poUcies  governing  the  production  and 
testing  of  weapon  systems,  particularly  those  deaUng  with  (1)  the 
purposes  of  lrip,  (2)  the  criteria  or  requirements  for  entering  lrip  and 
fuU-rate  production,  and  (3)  the  testing  requirements  related  to  this 
process.  We  used  the  results  of  our  extensive  body  of  work  from  the  past 
decade  or  so  on  defense  acquisition  programs  and  the  acquisition  process® 
to  determine  whether  the  lrip  concept,  as  currently  authorized  and 
practiced  by  dod,  has  resulted  in  a  premature  commitment  to  production 
of  both  mqjor  and  nonmqjor  systems.  We  reviewed  the  1993  report  of  the 
DOD-IG  on  LRIP  and  held  discussions  with  the  dod-ig  staff.  We  gathered  and 
summarized  data  on  numerous  ongoing  system  acquisition  programs  (both 
major  and  nonmajor  programs)  and  supplemented  that  information  with 
discussions  with  officials  from  the  Office  of  the  Secretary  of  Defense  and 
the  miUtary  services. 

In  addition,  we  gathered  and  analyzed  information  on  the  advantages  and 
disadvantages  of  conducting  ot&e  before  lrip  (for  both  major  and 
nonmqjor  systems).  We  also  held  discussions  with  those  officials  on  dod’s 
current  acquisition  strategies  and  ot&e  pohcies  and  practices. 

This  review  was  conducted  from  April  1993  to  May  1994  in  accordance 
with  generally  accepted  government  auditing  standards. 


'^See  the  Related  GAO  Products  section  of  this  report  for  a  partial  iisting  of  GAO  reports  on  these 
topics  since  late  1990.  Well  in  excess  of  100  GAO  reports,  covering  a  very  broad  range  of  programs, 
were  used  in  preparing  this  report. 
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DOD  Lacks  Controls  Over  LRIP 


Our  extensive  body  of  work  over  the  years  has  amply  demonstrated  that 
improper  usage  of  lrip  has  been  widespread.  Many  major  and  nonmajor 
systems  from  each  of  the  services  have  been  prematurely  committed  to 
production,  which  often  results  in  problems  being  found  after  a  substantial 
number  of  xinits  have  been  produced  and  a  significant  commitment  made 
to  the  entire  procurement  program.  In  addition,  contrary  to  the  statutory 
emphasis  on  minimum  lrip  quantities  and  conferee  statements,  many 
programs  continue  in  lrip  for  prolonged  periods,  dod’s  continuing 
reluctance  to  employ  the  discipline  of  early  ot&e  is  evident  in  each  of  the 
services  and  in  many  major  and  nonmajor  programs.  Adequate  controls 
have  not  been  estabhshed  over  the  start  and  continuation  of  lrip.  A 
requirement  to  successfully  complete  enough  independent  testing  in  an 
operational  environment  to  ensure  that  the  item  meets  its  key 
performance  parameters  before  lrip  starts  would  be  feasible  in  most  cases 
and  would  be  an  effective  management  control  over  the  premature  start  of 
production. 


Premature 
Commitments  to 
Production 


Over  the  years,  we  have  foimd  numerous  instances  from  all  three  services 
in  which  production  of  major  and  nonmqjor  systems  was  permitted  to 
begin  and  continue  based  not  on  the  systems’  technical  maturity,  but  on 
schedule  or  other  considerations,  dod  has  frequently  committed  programs 
to  production  without  assurance  that  the  systems  would  perform 
satisfactorily.  Many  of  the  weapon  systems  that  start  production 
prematurely  later  experience  significant  operational  effectiveness  and/or 
suitability  problems.  As  a  result,  mqjor  design  changes  were  often  needed 
to  correct  the  problems,  additional  testing  was  needed  to  verify  that  the 
corrective  action  was  effective,  and  costly  retrofits  were  needed  for  any 
delivered  units.  A  few  of  the  many  examples  of  premature  and  extensive 
commitments  to  production  of  major  and  nonmgjor  systems  are  shown  in 
the  following  tables.  Table  2.1  shows  systems  that  entered  lrip  before  any 
operational  tests  were  conducted  and  later  experienced  significant 
problems  during  the  tests.  Table  2.2  shows  systems  that  were  subjected  to 
early  operational  tests  but  were  allowed  to  enter  lrip  even  though  the 
performance  deficiencies  were  not  corrected. 
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Table  2.1:  Examples  of  Systems  That  Entered  LRIP  Before  Operational  Tests  Were  Conducted  and  Later  Experienced 
Problems  During  These  Tests 

Inadequate 


System 

Program 

category 

system 
deployed  to 
field 

Percent 
procured 
in  LRiP 

Comments 

Air  Force  C-17  Aircraft 

Major 

To  be 

determined 

33 

The  C-17's  reliability  is  significantly  less  than 
expected,  and  the  system  cannot  meet 
current  payload/range  specifications.  Also, 
while  known  problems  with  the  wings,  flaps, 
and  slats  are  being  fixed,  other  problems 
continue  to  emerge.  (GAO/T-NSIAD-94-166, 
Apr.  19,  1994.) 

Air  Force  AN/ALR-56C  Radar 
Warning  Receiver 

Nonmajor 

Yes 

8^ 

Despite  the  poor  QT&E  results,  the  Air  Force 
continued  full-rate  production  and  had 
acquired  about  750  systems  at  a  cost  of 
over  $570  million,  as  discussed  in  a 
classified  GAO  report. 

Air  Force  AN/ALQ-135  Quick 
Reaction  Capability  Jammer 

Nonmajor 

Yes 

100 

All  65  systems  were  produced  under  LRIP  at 
a  cost  of  $256  million,  before  any  OT&E  was 
conducted.  Because  of  performance 
problems,  most  of  the  jammers  were  placed 
in  storage  and  only  24  were  installed  on 
aircraft.  One  year  later,  the  24  jammers  were 
deactivated  because  of  poor  performance. 
(GAO/NSIAD-90-168,  July  11,  1990.) 

Air  Force  AN/ALQ-135  Improved 
Jammer 

Nonmajor 

Yes 

64‘’ 

Through  1993,  331  of  the  514  planned  units 
were  acquired  under  LRIP.  However,  the 
system  has  encountered  significant  software 
problems,  which  have  delayed  completion 
of  development  testing  by  about  2  years. 
OT&E  has  not  yet  started. 

Air  Force  AN/ALQ-131  Block  II 
Jammer 

Nonmajor 

Yes 

100 

After  the  Air  Force  bought  most  of  the  total 
quantity  of  units  under  LRIP,  tests  found 
serious  performance  problems.  As  a  result, 
the  system  was  deployed  with  the 
receiver/processor  inoperative  due  to  a  lack 
of  software.  Other  deficiencies  were  also 
present.  (GAO/NSIAD-90-168,  July  11, 

1990.) 

Air  Force  AN/USM-464  Electronic 
Warfare  Test  Set 

Nonmajor 

Yes 

100 

Before  the  Air  Force  conducted  OT&E,  72 
test  sets  were  procured  under  LRIP  at  a  cost 
of  $272  million.  Later  testing  showed  that  the 
equipment  would  not  meet  requirements, 
and  the  units  were  put  in  storage. 

(continued) 
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System 

Program 

category 

Inadequate 
system 
deployed  to 
field 

Percent 
procured 
in  LRIP 

Comments 

Air  Force  AN/ALQ-184  Jammer 

Nonmajor 

Yes 

8'= 

DOT&E  recommended  that  jammer 
production  be  stopped  because  of  poor 
OT&E  results.  However,  the  system  had 
already  entered  and  continued  full-rate 
production  anyway.  We  later  found  that  most 
of  the  24  jammers  deployed  to  a  tactical 
fighter  wing  had  been  placed  in  storage. 
(GAO/NSI AD-90-1 68,  July  11,  1990.) 

Navy  F-14D  Aircraft 

Major 

Yes 

100 

OT&E  showed  that  the  F-14D  was  not 
sufficiently  developed  and  lacked  critical 
hardware  and  sofWare  capabilities.  The 
program  was  terminated  after  55  units  were 
produced.  (GAO/IMTEC-92-21,  Apr.  2, 

1992.) 

Navy  T-45A  Aircraft 

Major 

Yes 

33 

One  year  into  LRIP,  OT&E  found  that  the 
T-45A  was  not  effective  in  a  carrier 
environment  and  was  not  operationally 
suitable  because  of  safety  deficiencies. 
Subsequent  major  design  changes  have 
included  a  new  engine,  new  wings,  and  a 
modified  rudder.  (GAO/NSI AD-91 -46,  Dec. 
14.  1990.) 

Navy  Pioneer  Unmanned  Aerial 
Vehicie 

Nonmajor 

Yes 

ula^ 

The  Navy  procured  and  deployed  Pioneer 
as  a  nondevelopmental  item  and  without 
testing  it.  Numerous  problems  ensued, 
including  engine  failures,  landing  difficulties, 
and  a  cumbersome  recovery  system.  Many 
modifications  were  required  to  bring  Pioneer 
up  to  a  minimum  essential  level  of 
performance. 

Army  Famiiy  of  Medium  Tacticai 
Vehicles 

Major 

To  be 
determined 

4® 

Before  the  Army  did  any  OT&E,  a  multiyear 
production  contract  was  awarded  for  up  to 
10,843  trucks.  Subsequent  OT&E  was 
suspended  because  the  vehicles  were 
found  to  be  unreliable  and  not  operationally 
effective.  However,  production  continues. 
(GAO/NSIAD-93-232.  Aug.  5,  1993.) 

Army  Palletized  Load 
System/Famiiy  of  Heavy  Tactical 
Vehicles 

Major 

Yes 

29 

OT&E  showed  the  system  to  be  not 
operationally  suitable.  Despite  the  need  for 
design  modifications  to  correct  reliability 
and  maintainability  problems,  full-rate 
production  was  approved. 

(Table  notes  on  next  page) 


Page  17 


GiAO/NSIAD-95-18  Weapons  Acquisition 


Chapter  2 

DOD  Lacks  Controls  Over  LRIP 


“Proceeded  beyond  LRIP  before  OT&E  was  conducted. 

•'Because  of  the  quantity  already  procured  in  LRIP  and  the  lack  of  OT&E  to  date,  additional  units 
are  likely  to  be  procured  in  LRIP. 

“Proceeded  beyond  LRIP  beyond  OT&E  was  conducted. 

“Production  was  not  separated  into  LRIP  and  full-rate  production  phases. 

“At  least  3,800  trucks  are  expected  to  be  produced  In  LRIP,  or  about  4  percent  of  the  more  than 
87,000  units  planned  to  be  procured. 
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Table  2.2:  Examples  of  Systems  That  Entered  LRIP  Despite  Problems  Identified  During  Early  Operational  Testing 

System 

Program 

category 

Inadequate 
system 
deployed  to 
field 

Percent 
procured  In 
LRIP 

Comments 

Army  AN/AVR-2,  AN/AVR-2A  Laser 
Warning  System 

Nonmajor 

Yes 

53® 

Even  though  over  900  units  have  been 
procured,  neither  the  redesigned  AN/AVR-2 
nor  the  further  redesigned  AN/AVR-2A  had 
completed  OT&E.  (GAO/NSI AD-93-1 4, 

Jan.  25,  1993.) 

ArmyAN/APR-39A(V)1  Radar 
Warning  Receiver 

Nonmajor 

Yes 

43“' 

In  1989,  the  Army’s  independent  testers 
recommended  curtailing  production 
because  of  poor  results.  However, 
production  continues. 

Navy  Airborne  Self-Protection 
Jammer 

Major 

N/A 

100® 

This  $2  billion  program  was  terminated  after 
procurement  of  95  systems  because  of 
failure  to  pass  required  OT&E. 

(GAO/NSI AD-92-1 03,  Mar.  23,  1992.) 

Navy  AN/SLQ-32  Shipboard 
Electronic  Countermeasures  System 

Major 

Yes 

73 

Numerous  modifications  have  been  made  at 
a  cost  of  over  $300  million,  but  the 
AN/SLQ-32’s  effectiveness  remains 
questionable.  (GAO/NSIAD-93-272,  Aug.  19, 
1993.) 

Navy  MK-50  Torpedo 

Major 

Yes 

100-^ 

OT&E,  conducted  after  LRIP  began,  was 
halted  until  technical  problems  were 
addressed.  However,  later  OT&E  continued 
to  show  performance  problems.  Due  to 
drastic  reductions  in  total  quantities,  all 

1 ,073  torpedoes  were  procured  in  LRIP. 

Navy  MH-53E  Helicopter 

Major 

Yes 

38 

OT&E  conducted  after  LRIP  started  revealed 
several  major  mechanical  deficiencies. 

Later,  OT&E  found  the  MH-53E  to  be 
marginally  operationally  effective  and  not 
operationally  suitable. 

Navy  AN/ALR-67(V)2  Radar 

Warning  Receiver 

Nonmajor 

Yes 

67 

OT&E  conducted  after  the  start  of  LRIP 
pointed  out  several  serious  problems. 
However,  by  that  time,  the  Navy  had  bought 
and  deployed  over  700  systems  at  a  cost  of 
$467  million.  The  Navy  later  spent 
$96  million  to  correct  the  deficiencies  and  to 
upgrade  the  systems.  However,  production 
of  the  upgrades  started  before  OT&E  was 
conducted,  and  many  units  were  placed  in 
storage  pending  completion  of  testing,  as 
discussed  in  a  classified  GAO  report. 

(continued) 
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System 

Program 

category 

Inadequate 
system 
deployed  to 
field 

Percent 
procured  in 
LRIP 

Comments 

Air  Force  AN/ALQ-131  Block  1 
Jammer 

Nonmajor 

Yes 

78 

The  Block  1  jammer  was  produced  and 
deployed  although  it  failed  to  pass  various 
reliability  and  maintainability  tests.  As  a 
result,  many  of  the  jammers  required  major 
part  replacements  and  technical 
adjustments,  as  discussed  in  a  classified 
GAO  report. 

“Quantity  includes  some  AN/AVR-2  full-rate  production  units. 

'^Production  was  not  separated  into  LRIP  and  full-rate  production  phases.  However,  2,000  units 
(or  43  percent  of  the  total  program  quantity  of  4,604  units)  were  on  contract  at  the  time  of  the 
independent  tester's  recommendation. 

“Program  terminated  while  in  LRIP. 

“Due  to  drastic  reductions  in  total  procurement  quantities  of  the  MK  50,  the  Navy  decided  not  to 
get  approval  for  full-rate  production. 


Major  Production 
Commitments  Are 
Made  During  LRIP 


Programs  that  enter  production  prematurely  often  require  more  time  and 
resources  than  originally  planned  to  correct  problems  and  to  meet  the 
requirements  for  full-rate  production.  LRIP  is  often  continued,  despite  the 
evidence  of  technical  problems,  weU  beyond  that  needed  to  provide  test 
articles  and  to  establish  an  initial  production  capability.  As  a  result,  mgjor 
production  commitments  are  often  made  during  lrip. 

In  the  conference  report  for  the  lrip  legislation,  the  conferees  stated  that 
they  did  not  intend  to  authorize  the  continuance  of  lrip  on  an  indefinite 
basis.  Nevertheless,  the  existing  lrip  legislation  does  not  include  any 
specific  principles  or  guidelines  on  when  and  how  programs  should  begin 
LRIP,  on  the  type  and  amoimt  of  testing  to  be  done  before  lrip,  on  how 
much  LRIP  can  or  should  be  done,  or  under  what  circumstances  lrip  should 
be  curtailed  or  stopped.  Instead,  the  emphasis  has  been  placed  almost 
entirely  on  the  full-rate  production  decision,  at  which  point  the  law 
requires,  among  other  things,  that  a  report  be  provided  on  the  adequacy  of 
the  testing  conducted  and  an  assessment  be  made  of  the  system’s 
operational  effectiveness  and  suitability.  Although  programs  are  delayed 
getting  approval  for  full-rate  production,  lrip  is  rarely  stopped  or  slowed 
significantly.  As  a  result,  the  decision  to  start  lrip,  in  many  cases,  is  also 
the  de-facto  full-rate  production  decision. 
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DOD  Fails  to 
Emphasize  Early 
OT&E 


Don’s  written  policies  provide  that  acquisition  strategies  be  event-driven 
and  link  major  contractual  commitments  and  milestone  decisions  to 
demonstrated  accomplishments  in  development,  test,  and  initial 
production.  However,  dod  pohcies  state  that  a  primary  goal  in  developing 
an  acquisition  strategy  shall  be  to  minimize  the  time  and  cost  of  satisfying 
a  need  consistent  with  common  sense  and  sound  business  practices. 

In  addition,  dod’s  policies  state,  but  without  detailed  requirements,  that 
OT&E  should  be  conducted  throughout  the  acquisition  process.  However, 
while  DOD  is  statutorily  required  to  conduct  ot&e  before  fuU-rate 
production  is  jqiproved,  dod’s  pohcies  permit  lrip  to  begin  before  any  OT&E 
is  conducted.  The  point  at  which  lrip  begins  is  not  a  required  milestone 
under  dod  pohcy. 

As  a  result,  for  many  mqjor  defense  acquisition  programs,  the  services  do 
not  plan  to  conduct  any  OT&E  prior  to  the  start  of  lrip.  It  has  been  and 
continues  to  be  the  exception,  rather  than  the  rule,  for  programs  to 
include  ot&e  before  lrip  starts.  In  some  instances,  the  services  plan  to  start 
LRIP  even  though  they  plan  to  use  developmental  or  prototype  units  for 
their  initial  ot&e,  not  lrip  units.  Although  not  required  by  written  dod  or 
Navy  pohcy,  the  Navy  now  performs  a  hmited  phase  of  ot&e  before  lrip  to 
prepare  for  later  phases  of  ot&e  on  some  of  its  programs.  However,  these 
programs  are  not  required  to  meet  specific  testing-related  criteria  before 
entering  lrip.  As  shown  in  table  2.2,  even  when  some  ot&e  was  conducted 
prior  to  the  start-up  of  production,  identified  problems  were  not  verified 
as  corrected,  and  significant  performance  problems  emerged  later  in  the 
program. 

Over  the  past  several  years,  dod  has  stated  that  it  planned  to  reemphasize 
the  need  for  ot&e  as  early  as  possible  in  the  acquisition  process.  However, 
we  have  not  detected  any  reemphasis  on  early  ot&e,  and  dod’s  1991 
revision  of  its  key  acquisition  directives  did  not  address  this  issue,  dod 
acquisition  and  testing  officials  concede  that  there  has  not  been  any  mqjor 
reemphasis  on  early  ot&e.  In  fact,  dod  has  recently  supported  legislative 
proposals  that  would  reduce  the  current  overall  requirements  to  conduct 
OT&E.^ 

DOD  has  recognized  that  reducing  the  amount  of  production  prior  to 
completing  development  provides  for  greater  design  maturity,  which 
increases  the  likelihood  of  meeting  system  requirements  and  avoiding 


'Acquisition  Reform:  Role  of  Test  and  Evaluation  in  System  Acquisition  Should  Not  Be  Weakened 
(GAO/T-NSIAD-94-124,  Mar.  22,  1994). 
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retrofit  costs.  In  commenting  on  our  1992  report,^  dod  officials  said  they 
were  lessening  the  amount  of  concurrent  development  and  production  in 
weapon  programs  due  to  the  end  of  the  Cold  War.  In  1992,  the  Under 
Secretary  of  Defense  for  Acquisition  also  stated  that  the  need  to  replace 
existing  weapon  systems  in  order  to  maintain  a  significant  technological 
advantage  was  no  longer  as  urgent.  However,  cicquisition  strategies  of 
many  current  programs  do  not  reflect  these  positions,  dod’s  acquisition 
practices  continue  to  stress  the  importance  of  minimizing  the  time  to 
deploy  new  or  improved  weapon  systems. 

Highly  concurrent  acquisition  strategies  continue  to  be  featured  in  many 
current  mgjor  and  nonm^jor  programs,^  with  little,  if  any,  OT&E  expected 
until  well  after  the  start  of  production  and  a  significant  commitment  is 
made  to  the  procurement  of  the  system.  Our  analysis  of  the  current 
selected  acquisition  reports  shows  that  many  programs  continue  to 
postpone  initial  OT&e  until  well  after  the  start  of  production. 

•  LRIP  is  expected  to  be  approved  in  February  1996  for  the  Army’s  Secure 
Mobile  Anti-Jam  Reliable  Tactical  Terminal.  Initial  ot&e  will  not  be 
completed  until  July  1998,  by  which  time  a  total  of  125  units,  or  3  years  of 
LRIP,  is  planned  to  be  approved  out  of  a  total  program  quantity  of  367  units. 

•  The  LRIP  decision  for  the  Air  Force’s  F-22  aircraft  program  is  expected  in 
June  1998,  and  initial  ot&e  is  to  be  conducted  from  March  to 
November  2001.  Thus,  1  year  of  preproduction  verification  and  4  years  of 
LRIP — 80  aircraft  out  of  a  total  quantity  of  442  units — are  planned  to  be 
approved  before  completion  of  ot&e. 

•  The  Navy  plans  to  procure  106  of  the  630  planned  Multifunctional 
Information  Distribution  Systems  before  OT&E  is  completed  in  December 
2000  and  a  fuU-rate  production  decision  is  made  in  June  2001.  In  addition, 
42  prototype  systems  are  to  be  built  as  part  of  the  system  development 
effort. 

These  programs  feature  m^jor  commitments  to  lrip  before  development  is 
completed  and  before  any  ot&e  is  completed,  even  though  developmental 
prototypes  are  expected  to  be  available  for  testing  in  these  programs. 
Accordingly,  a  substantial  and  frequently  irreversible  commitment  to 
production  will  have  been  made  before  the  results  of  independent  testing 
are  available  to  decisionmakers. 


^Weapons  Acquisition:  A  Rare  Opportunity  for  Lasting  Change  (GAO/NSIAD-93-16,  Dec.  1992). 

program  with  high  levels  of  concurrency  typically  proceeds  into  production  before  a  significant 
amount  of  initial  OT&E  is  conducted. 
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Controls 

Recommended  Over 
the  Start  and 
Continuation  of  LRIP 


In  its  1993  report,^  the  dod-ig  found  that  mgjor  defense  acquisition 
programs  were  entering  lrip  without  meeting  development,  testing,  and 
production  readiness  prerequisites.  As  a  result,  the  dod-ig  concluded  that 
DOD  incurred  excessive  program  risk  of  overcommitment  to  production  of 
systems  without  obtaining  assurance  that  the  design  is  stable,  potentially 
operationally  acceptable,  and  capable  of  being  produced  efficiently. 
Among  other  things,  the  dod-ig  recommended  that  dod  (1)  provide 
guidance  on  the  specific  minimum  required  program  accomplishments  for 
entry  into  and  continuation  of  lrip  and  (2)  require  that  program-specific 
exit  criteria  be  established  for  entry  into  and  continuation  of  lrip.  dod  is 
currently  considering  what,  if  any,  actions  will  be  taken  in  light  of  the 
dod-ig’s  recommendations. 


'‘I/)w-Rate  Initial  Production  in  Msyor  Defense  Acquisition  Programs  (DOD  Inspector  General  Report 
No.  94-014,  Nov.  9, 1993). 
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Conclusions 


The  decision  to  begin  lrip  should  be  given  much  more  attention  because 
decisionmakers  find  it  very  difficult  to  stop  or  slow  down  programs  once 
they  are  in  production.  Given  the  cost  risks  involved  and  dod’s  inability  or 
unwillingness  to  curtaQ  production  after  it  starts,  we  agree  with  the  dod-ig 
that  controls  are  urgently  needed  over  the  start  and  continuation  of  lrip. 

A  key  criterion  for  all  programs  beginning  lrip  should  be  the  completion  of 
a  phase  of  independent  testing  in  an  operational  environment.  During  such 
testing,  some  problems  should  be  expected.  However,  enough  realistic 
testing  should  be  conducted  for  the  services’  independent  testing  agencies 
and/or  dot&e  to  be  able  to  certify  to  the  decision  authority  that  (1)  the 
system’s  developmental  testing  is  essentially  complete  and  the  basic 
results  have  been  validated  in  an  operational  environment,  (2)  the  system 
has  clearly  shown  that  it  can  meet  the  key  parameters  among  its  minimum 
acceptable  performance  requirements,^  (3)  the  system  has  clearly 
demonstrated  the  potential  to  fuUy  meet  all  of  its  minimum  acceptable 
requirements  for  performance  and  suitability  without  mqjor  or  costly 
design  changes,  and  (4)  the  system  should  be  able  to  readily  complete  its 
remaining  ot&e  in  time  to  support  the  planned  full-rate  production 
decision.^  Comprehensive  testing  of  a  system’s  operational  suitability 
features,  such  as  supportability,  may  not  be  possible  during  early 
independent  testing.  However,  the  testing  should  be  sufficient  to  reveal 
major  suitability  problems. 

Conducting  ot&e  before  lrip  will  not,  by  itself,  result  in  a  better  weapon 
system,  but  it  is  the  best  means  available  to  guard  against  the  premature 
start  of  production.  Decisionmakers  need  verifiable  information  on  system 
design  maturity  and  where  corrective  actions  are  needed  before 
production  start-up.  Every  effort  should  be  made  to  correct  problems  in 
development,  not  in  production,  because  early  fixes  are  less  expensive, 
easier  to  implement,  and  less  disruptive.  In  today’s  national  security 
environment,  there  should  be  very  few  cases  in  which  an  urgent  need 
dictates  that  dod  start  production  without  assurance  that  the  system  will 
work  as  intended. 


'Minimum  acceptable  requirements  are  the  values  for  the  particular  parameters  that  are  required  to 
provide  a  system  capability  that  will  satisfy  the  validated  mission  need.  These  requirements  are  also 
known  as  thresholds  and  are  established  in  the  Operational  Requirements  Document  at  each 
milestone.  Key  parameters  are  those  for  which  the  decision  authority  would  require  a  reevaluation  of 
alternative  concepts  or  design  sqrproaches  if  the  thresholds  are  not  met. 

^This  certification  should  be  made  by  DOT&E  for  all  major  defense  acquisition  programs  and  by  the 
services’  independent  testing  agencies  for  all  other  systems. 
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We  realize  that,  for  some  programs,  a  significant  effort  (personnel  and 
facilities)  may  be  needed  to  produce  one  or  more  prototypes  for  a  phase  of 
early  ot&e.  These  programs  would  typically  involve  inherent  fabrication 
complexity,  small  procurement  quantities,  high  unit  cost,  and  long  unit 
production  periods.  To  suspend  that  type  of  effort  while  ot&e  is  underway 
could  be  costly  and  disruptive.  Alternatively,  key  subsystems  should  be 
independently  tested  on  surrogate  platforms  before  production.  Once 
underway,  production  should  be  limited  until  acceptable  ot&e  results  are 
obtained  on  the  entire  system. 

We  beheve  that  lrip  should  be  used  to  focus  on  (1)  addressing 
producibility  and  product  quality  issues;  (2)  producing  just  enough 
systems  to  support  initial  ot&e,  to  prove  out  the  production  process,  and 
to  sustain  the  production  Une;  and  (3)  testing  those  systems  and  correcting 
any  deficiencies.  A  limit  on  the  quantity  that  can  be  produced  under  lrip 
would  provide  an  opportunity  to  correct  problems  that  are  identified 
during  initial  ot&e,  without  incurring  the  risk  of  overproducing  under  the 
LRIP  phase. 


Recommendations 


We  recommend  that  the  Secretary  of  Defense  revise  dod’s  acquisition 
policies  in  the  following  ways: 

Require  that,  before  entry  into  lrip,  programs  (with  the  exception  of  ships, 
satellites,  and  those  other  programs  that  involve  inherent  fabrication 
complexity,  small  procurement  quantities,  high  unit  costs,  and  long  unit 
production  periods)  plan,  buy  prototypes  for,  and  conduct  enough  realistic 
testing  for  the  service’s  independent  testing  agency  and/or  dot&e  to  be 
able  to  certify  to  the  decision  authority  that  (1)  the  system’s 
developmental  testing  is  essentially  complete  and  the  basic  results  of  that 
testing  have  been  validated  in  an  operational  environment;  (2)  the  system 
has  clearly  shown  that  it  can  meet  the  key  parameters  among  its  minimum 
acceptable  performance  requirements;  (3)  the  system  has  clearly 
demonstrated  the  potential  to  fidly  meet  all  of  its  minimum  acceptable 
requirements  for  performance  and  suitability  without  mqjor  or  costly 
design  changes;  and  (4)  the  system  should  be  able  to  readily  complete  its 
remaining  ot&e  in  time  to  support  the  planned  fiiU-rate  production 
decision. 

Require  that  those  programs  excluded  from  the  requirement  to  test 
prototypes  instead  test  all  key  subsystems  in  an  operational  environment 
before  entry  into  lrip. 
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•  Adopt  the  recommendations  made  by  the  dod-ig  regarding  controls  over 
the  start  and  continuation  of  lrip  such  as  (1)  providing  guidance  on  the 
specific  minimum  required  program  accomplishments  for  entry  into  and 
continuation  of  lrip  and  (2)  requiring  that  program-specific  exit  criteria  be 
established  for  entry  into  and  continuation  of  lrip. 

We  also  recommend  that  the  Secretary  of  Defense  work  with  the  service 
secretaries  to  ensme  that  these  policies  are  implemented  for  the 
acquisition  of  both  mqjor  and  nonmajor  systems. 


Matters  for 

Congressional 

Consideration 


The  legislation  defining  lrip  has  not  been  effective  in  accomplishing  its 
purpose,  which  was  to  limit  the  commitment  to  mqjor  production 
quantities  pending  satisfactory  completion  of  ot&e.  Therefore,  we 
recommend  that  the  Congress  legislatively  mandate  (1)  that  certain  ot&e 
requirements  be  met  before  lrip  may  start  and  (2)  specific  limits  on  the 
number  of  units  allowed  to  be  produced  during  lrip.  Specifically,  the 
Congress  may  wish  to  require  that  aU  defense  acquisition  programs  (mqjor 
and  nonmajor)  conduct  enough  realistic  testing  on  the  entire  system  or 
key  subsystems  to  ensure  that  its  key  performance  parameters  are  met 
before  lrip  is  permitted  to  start.  In  addition,  the  Congress  may  wish  to 

(1)  specify  a  percentage  (10  percent,  for  example)  of  a  system’s  total 
procurement  beyond  which  a  program  may  not  proceed  during  lrip  and/or 

(2)  amend  10  U.S.C.  2400  (by  deleting  subsection  (b)(3))  to  preclude  the 
use  of  LRIP  authority  to  ramp-up  the  production  rate  prior  to  the  successful 
completion  of  ot&e. 
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Note:  GAO  comments 
supplementing  those  in  the 
report  text  appear  at  the 
end  of  this  appendix. 


See  comment  1. 


See  comment  2. 


See  comment  3. 


See  comment  4. 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHINGTON  DC  203013000 


ACOUISmON  AND  \  )  AUG  1994 

TECHNOLOGY 


Mr.  Frank  C.  Conahan 
Assistant  Comptroller  General 
National  Security  and  International 
Affairs  Division 

U.S.  General  Accounting  Office 
Washington,  DC  20548 

Dear  Mr.  Conahan: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  General  Accounting  Office 
(GAO)  draft  report,  'WEAPONS  ACQUISITION:  Low-Rate  Initial  Production  Used  to  Buy 
Weapon  Systems  Prematurely,”  dated  June  28, 1994  (GAO  Code  707065),  OSD  Case  9725. 
DoD  partially  concurs  with  the  report  findings,  but  non-concurs  with  all  but  one  of  the 
recommendations. 

While  much  of  ttie  information  reported  by  the  GAO  is  factual,  some  of  the  implications 
to  be  drawn  from  the  information  are  misleading.  For  example,  Table  2-1  lists  the  percentage 
of  units  procured  in  Low-Rate  Initial  Production  (LRIP).  The  C-17  is  listed  at  33%.  That 
number  is  true  based  on  a  buy  of  120  airplanes.  However,  the  original  program  was  for  200 
plus  airplanes.  The  number  of  airplanes  to  be  procured  was  reduced  as  a  result  of  problems 
encountered  in  testing.  If  the  number  of  units  procured  in  LRIP  Is  compared  to  the  total 
planned  program,  the  percentage  would  be  16%  rather  than  33%.  In  addition,  AN/ALO-32 
shows  a  100%  buy  in  LRIP.  However,  the  supporting  narrative  states  that  only  65  systems 
were  procured  because  DoD  testing  found  problems.  The  GAO  report  fails  to  mention  the  total 
planned  buy.  This  same  problem  exists  lor  the  AN/USM-464,  AN/ALQ-131.  F-14D.  and  the 
Airborne  Sell-Protection  Jammer.  Lastly,  the  GAO  report  cites  the  Navy  Pioneer  Unmanned 
Aerial  Vehicle  as  a  system  that  entered  LRIP  before  conducting  any  operational  testing  and 
later  experienced  significant  problems.  The  Pioneer  system  was  procured  under  special 
circumstances,  but  was  not  an  LRIP  effort. 

The  Department  also  questions  the  validity  of  the  report  findings  for  three  reasons. 

First,  the  GAO  methodology  is  such  that  the  magnitude  of  the  problem  you  present  is  undear. 
There  is  no  indication  of  the  number  of  programs  with  Low-Rate  Initial  Production  (LRIP)  in 
DoD.  Therefore,  objective  readers  of  the  report  have  no  way  to  determine  whether  the  20 
programs  cited  constitute  a  majority  or  small  minority  of  LRIP  programs.  Second,  the 
programs  chosen  do  not  appear  to  be  randomly  selected.  Rather,  they  are  programs  that 
GAO  has  previously  reviewed  and  are  heavily  electronic  oriented.  Third,  the  report  contains 
several  sweeping  generalizations.  For  example,  GAO  asserts  that  "DoD's  acquisition  policies 
permit  LRIP  to  start  without  doing  any  OT&E.”  GAO  also  assorts  that,  there  are  no  clear 
principles  or  guidance  -  in  legislation  or  DoD  policy  -  on  when  or  how  programs  should  begin 
LRIP...."  In  fact,  LRIP  guidance  is  cited  in  over  90  places  in  DoDD  5000.1 .  DoDI  5000.2,  and 
DoD  5000.2-M.  Specifically.  DoDI  5000.2,  Part  8  notes  the  importance  of  the  early 
involvement  of  both  developmental  and  operational  testers  in,and  the  program  schedule  chart 


o 
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See  comment  3. 


See  comment  5. 


See  comment  6. 


See  comment  7. 


See  comment  8. 


in  DoD  5000.8-M,  Section  4-B  shows  operational  lest  of  engineering  development  models  prior 
to  the  beginning  of  LRIP. 

It  is  the  DoD  position  that  because  of  the  GAO  lack  of  familiarity  with  the  process,  and 
the  attempt  to  extrapolate  from  a  very  small  sample,  GAO  report  conclusions  and 
recommendations  are  seriously  flawed.  The  GAO  recommendations  to  do  more  full-up  testing 
would  have  the  effect  of  adding  time  and  cost  to  weapons  system  development  without 
improving  the  identification  of  the  majority  of  problems  found  in  operational  testing.  DoD  has 
been  using  operational  assessments,  rather  than  full-up  testing,  prior  to  LRIP  to  identify 
problems  that  could  lead  to  redesign. 

Also,  DoD  strenuously  objects  to  the  matters  for  congressional  consideration 
concerning  additional  legislation.  It  is  the  DoD  position  that  current  legislation  (10  U.S.C. 
§2400)  provides  sufficient  statutory  guidance  to  limit  the  production  of  LRIP  units  without 
potentially  causing  costly  and  disruptive  gaps  in  production.  In  addition,  the  Department 
opposes  the  mandating  of  OT&E  requirements  that  must  be  met  before  the  start  of  production 
and  fixed  percentage  limits  on  the  number  of  units  that  may  be  produced  In  LRIP.  Both 
provisions  could  cause  production  stretch-outs  and  production  stoppages.  DoD  also  opposes 
changes  to  §2400  that  would  preclude  the  use  of  LRIP  authority  to  efficiently  ramp-up  the 
production  rate.  Such  changes  would  most  certainly  stretch-out  production  programs. 

In  conclusion,  the  Department  agrees  that  there  are  problems  with  the  current  statutory 
regime  for  testing,  and  supports  constructive  changes  in  the  current  testing  statutes. 

Examples  of  the  kinds  of  problems  and  the  kinds  of  statutory  changes  that  are  necessary  to 
correct  the  problems  are  contained  In  the  testimony  presented  by  the  Deputy  Linder  Secretary 
of  Defense  (Acquisition  Reform)  before  the  Senate  Federal  Services,  Post  Office,  and  Civil 
Service  Subcommittee,  on  March  22, 1994. 

Detailed  DoD  comments  on  the  GAO  report  recommendations  and  matters  for 
congressional  consideration  are  contained  in  Enclosure  1 .  A  copy  of  the  DoD  response  to  the 
related  DoDIG  report  is  contained  in  Enclosure  2.  A  copy  of  the  Deputy  Under  Secretaiy*s 
testimony  is  contained  in  Enclosure  3.  DoD  appreciates  the  opportunity  to  comment  on  the 
draft  GAO  report. 

Sincerely, 

Colleen  A.  Preston 

Deputy  Under  Secretary  of  Defense 
(Acquisition  Reform) 

Enclosures; 

As  Stated 
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GENERAL  ACCOtIHTING  OFFICE  DRAFT  REPORT  -  DATED  JDNE  28,  1994 
(GAO  CODE  707065)  OSD  CASE  9725 

"WEAPONS  ACQUISITION:  LOW-RATE  INITIAL  PRODUCTION 
USED  TO  BUY  WEAPON  SYSTEMS  PREMATURELY" 

DEPARTMENT  OF  DEFENSE  COMMENTS 


RECOMMENDATIONS 


o  RECOMMENDATION  1:  The  GAO  recommended  that  the  Secretary 

of  Defense  revise  DoD  acquisition  policies  to  require  that, 
before  entry  into  low-rate  initial  production,  programs 
plan,  buy  prototypes  for,  and  conduct  enough  realistic 
testing  for  the  Service's  independent  testing  agency  and/or 
the  Director,  Operational  Test  and  Evaluation  (DOTiE)  to  be 
able  to  certify  to  the  decision  authority  that  (1)  system's 
developmental  testing  is  essentially  complete  and  the 
basic  results  of  that  testing  have  been  validated  in  an 
operational  environment,  (2)  the  system  has  clearly  shown 
that  it  can  meet  the  key  parameters  among  its  minimum 
acceptable  performance  requirements,  (3)  the  system  has 
clearly  demonstrated  the  potential  to  fully  meet  all  of 
its  minimum  acceptable  requirements  for  performance  and 
suitability  without  major  or  costly  design  changes, 
and  (4)  the  system  should  be  able  to  readily  complete 
its  remaining  OTSE  in  time  to  support  the  planned  full-rate 
production  decision.  (pp.  4-5,  p.  32/GAO  Draft  Report) 

DOD  RESPONSE:  Non  concur.  The  GAO  based  this  recommenda¬ 
tion  on  two  false  assumptions — developmental  test  agencies 
are  not  independent  of  the  system  developer  and  the  DoD  only 
does  developmental  testing  prior  to  LRIP.  It  is  the  DoD 
position  that  neither  assumption  is  valid.  First,  in  each 
of  the  Military  Services,  developmental  test  agencies  are 
See  comment  9.  independent  of  system  developers.  For  example,  in  the  Army 

developmental  test  is  conducted  by  the  Test  and  Evaluation 
Command  (TECOM) .  Development  of  major  systems  is  done  by 
Program  Managers  who  report  to  Program  Executive  Officers 
(PEO) .  Nonmajor  systems  are  developed  by  Program  Managers 
who  report  to  the  commanders  of  systems  commands  (e.g. ,  Tank 
Automotive  Command  (TACOM)  or  the  Missile  Command  (MICOM) . 
The  TECOM  is  not  under  the  command  of  either  the  systems 
commanders  or  the  PEOs.  Second,  DoD  currently  uses 
operational  assessments  as  a  means  of  determining 
operational  problems  that  could  lead  to  redesign.  These 
See  comment  10.  operational  assessments  occur  at  various  times  in  the 
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See  comment  1 1 . 


See  comment  12. 


See  comment  13. 


See  comment  14. 


developmental  cycle.  The  DoD  policy  supports  and  encourages 
operational  assessments  as  early  in  the  development  of  a 
system  as  is  practical  and  as  often  as  is  necessary  to 
assist  in  the  design  of  the  system. 

In  addition,  DoD  policy  (DoDI  5000.2,  Part  8)  currently 
requires  that  the  developing  agency  formally  certify  that  a 
system  is  ready  for  dedicated  operational  test.  It  is  the 
DoD  position  that  this  recommendation  does  not  add  anything 
to  the  current  policy  but  rather  appears  to  be  a  new 
requirement  for  operational  testing  prior  to  LRIP.  Such 
additional  testing  would  add  cost  and  time  to  weapon  system. 
It  would  also  most  probably  result  in  a  new  milestone 
decision  review  to  enter  LRIP.  A  new  milestone  would  add 
administrative  burden  to  the  program  manager  in  preparing 
for  the  new  milestone  decision  review.  This  certainly  is 
not  consistent  with  the  Administration  or  the  Congress  goal 
of  streamlining  the  acquisition  process  and  reducing  the 
acquisition  work  force. 

Finally,  it  is  the  DoD  position  that  expecting  a  non- 
production  prototype  to  meet  all  key  parameters  before  LRIP 
is  unreasonable.  A  fundamental  principle  of  operational 
testing  is  that  production-representative  articles  be  used 
as  much  as  possible  in  as  realistic  an  environment  as 
possible.  Since  production-representative  articles  are 
usually  obtained  through  LRIP,  it  does  not  follow  that 
realistic  OT&E  can  always  be  done  before  LRIP.  The  use 
of  developmental  prototypes  to  accomplish  the  OTSE  objective 
is  questionable. 


o  RECOMMEMDATION  2;  The  GAO  recommended  that  the  Secretary  of 
Defense  revise  the  DoD  acquisition  policies  to  require  that, 
those  programs  excluded  from  the  requirement  to  test  proto¬ 
types  instead  test  all  key  subsystems  in  an  operational 
environment  before  entry  into  low-rate  initial  production, 
(p.  5,  p.  32/GAO  Draft  Report) 

DOD  RESPOM8E:  Non  concur.  It  is  the  DoD  position  that 
the  majority  of  problems  found  in  operational  test  are  the 
result  of  integration  and  interface  problems  among  sub¬ 
systems.  Using  independent  tests  on  surrogate  platforms 
would  not  generally  find  these  types  of  problems  nor 
mitigate  redesign  in  IRIP.  Determining  appropriate 
surrogates  and  providing  the  engineering  required  to 
support  a  "foreign"  subsystem  would  add  time  and  costs 
to  the  program  with  negligible  benefits. 
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See  comment  15. 


See  comment  8. 


See  comment  7. 


O  RECOMMENDATION  3:  The  GAO  recommended  that  the  Secretary 
of  Defense  adopt  the  recommendations  made  by  the  DoD 
Inspector  General  regarding  controls  over  the  start  and 
continuation  of  low-rate  initial  production  such  as  (1) 
providing  guidance  on  the  specific  minimum  required  program 
accomplishments  for  entry  into  and  continuation  of 
low-rate  initial  production  and  (2)  requiring  that 
program-specific  exit  criteria  be  established  for  entry 
into  and  continuation  of  low-rate  initial  production. 

(p.  5,  p.  3VGAO  Draft  Report) 

POD  RESPONSE:  Partially  concur.  The  Office  of  the  Under 
Secretary  of  Defense  for  Acquisition  and  Technology 
(USO(A&T))  has  taken  the  position  for  DoD  that  there  is 
benefit  to  supplementing  current  policy  regarding  the 
transition  of  major  defense  acquisition  programs  into  LRIP. 
However,  as  discussed  in  our  May  5,  1994  and  January  12, 
1994,  correspondence  with  the  DoDIG,  the  USD(A4T))  is 
working  closely  with  the  OSD  and  Joint  Staffs  and  the 
Military  Departments  to  develop  policy  guidance.  The  DoD 
has  established  a  Process  Action  Team  (PAT)  on  the  Defense 
Acquisition  Board  process  to  address  this  issue.  It  is  the 
DoD  position  that  a  reasonable  approach  is  to  await  the 
findings  and  recommendations  of  the  PAT,  before  issuing 
policy  guidance  that  could  be  subject  to  change  through  PAT 
recommendations.  The  PAT  report  is  expected  within  90  to 
120  days.  The  DoD  May  5  and  January  12  responses  to  DoDIG 
Report  94-014  are  enclosed. 


o  RECOMMENDATION  4 :  The  GAO  recommended  that  the  Secretary  of 
Defense  work  with  the  Service  secretaries  to  ensure  that 
these  policies  are  implemented  for  the  acquisition  of  both 
major  and  nonmajor  systems  of  low-rate  initial  production. 

(p.  5,  p.  33/GAO  Draft  Report) 

DOD  RESPONSE:  Non-concur.  The  Secretary  of  Defense  and  the 
Under  Secretary  of  Defense  (Acquisition  and  Technology)  are 
working  with  the  Service  Secretaries  and  the  Service 
Acquisition  Executives  to  re-engineer  the  current 
acquisition  process  to  maintain  the  U.S.  technological 
superiority  and  a  strong,  globally  competitive  national 
industrial  base,  while  reducing  acquisition  costs  and 
protecting  the  public  trust.  Part  of  that  reengineering 
effort  is  the  identification  and  implementation  of  those  DoD 
acquisition  policies  that  contribute  to  the  achievement  of 
this  goal  and  the  cancellation  of  those  acquisition  policies 
that  do  not  contribute  to  the  achievement  of  this  goal.  Not 
all  policies  should  be  implemented  in  the  reengineered 
systems  acquisition  process. 
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See  comments  5  and  6. 


See  comment  6. 


MATTERS  FOR  COHGRESSIOMAL  CONSIDERATION 


o  snsflESTlON  l!  The  GAO  suggested  that  the  Congress  nay  wish 
to  legislatively  mandate  (1)  that  certain  operational  test 
and  evaluation  requirements  be  met  before  low-rate  initial 
production  may  start  and  (2)  specific  limits  on  the  number 
of  units  allowed  to  be  produced  during  low-rate  initial 
production.  (p.  5,  p.  33/GAO  Draft  Report) 

POD  RESPONSE:  Non  concur.  It  is  the  DoD  position  that 
current  legislation  (10  U.S.C.  2400)  provides  sufficient 
statutory  guidance  to  limit  the  production  of  LRIP  units 
without  potentially  causing  costly  and  disruptive  gaps  in 
production.  In  addition,  the  Department  opposes  mandating 
OT&E  requirements  that  must  be  met  before  the  start  of 
production  and  fixed  percentage  limits  on  the  number  of 
units  that  may  be  produced  in  LRIP.  Both  provisions  could 
cause  production  stretch-outs  and  production  stoppages. 


o  SPGGEBTION  2:  The  GAO  suggested  the  Congress  may  wish  to 
require  that  all  defense  acquisition  programs  (major  and 
nonmajor)  conduct  enough  realistic  testing  on  ^e  entire 
system  or  key  subsystems  to  assure  that  its  key  performance 
parameters  are  met  before  low-rate  initial  production  is 
permitted  to  start.  (p.  5,  p.  33/GAO  Draft  Report) 

DOD  RESPONSE:  Non  concur.  As  discussed  in  Recommendation 
2,  the  majority  of  problems  found  in  operational  test  are 
the  result  of  integration  and  Interface  problems  among 
subsystems.  Using  independent  tests  on  surrogate  platforms 
would  not  generally  find  these  types  of  problems  nor 
mitigate  redesign  in  LRIP.  Determining  appropriate 
surrogates  and  providing  the  engineering  required  to  support 
a  'foreign"  subsystem  would  add  time  and  costs  to  the 
program  with  negligible  benefits.  The  Department  opposes 
mandating  OT&E  requirements  that  must  be  met  before  the 
start  of  production  on  the  number  of  units  that  may  be 
produced  in  LRIP.  Such  a  provision  could  cause  production 
stretch-outs  and  production  stoppages. 
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See  comments  6  and  16. 


o  SDoaESTIOK  3:  The  GAO  suggested  the  Congress  may  wish  to 
(1)  specify  a  percentage  (10  percent,  for  example)  of  a 
system's  total  procurement  beyond  which  a  program  may  not 
proceed  during  low-rate  initial  production  and/or  (2)  amend 
10  U.S.C.  2400  by  deleting  subsection  (b) (3)  to  preclude 
the  use  of  low-rate  initial  production  authority  to  ramp-up 
the  production  rate  prior  to  the  successful  completion 
of  OT4E.  (p.  5,  p.  33/GAO  Draft  Report) 

POD  RESPOMSB:  Non  concur.  The  Department  opposes  mandating 
fixed  percentage  limits  on  the  number  of  units  that  may  be 
produced  in  LRIP.  Both  provisions  could  cause  production 
stretch-outs  and  production  stoppages.  The  DoD  also  oppose 
changes  to  2400  that  would  preclude  the  use  of  LRIP 
authority  to  efficiently  ramp-up  the  production  rate.  Such 
changes  would  most  certainly  stretch-out  production 
programs. 
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GAO  Comments 


1.  The  total  procurement  quantity  of  C-17s  was  reduced  for  several 
reasons,  including  affordability,  but  we  strongly  dispute  the  Department  of 
Defense’s  (dod)  assertion  here  that  the  reduction  was  as  a  result  of 
problems  encountered  in  testing. 

2.  After  careful  review,  we  have  concluded  that  the  percentages  in 
table  2.1  are  correct  as  stated  and  that  no  changes  are  needed.  Our  report 
shows  that  73  percent  of  the  AN/SLQ-32  program  quantity  was  bought 
during  lrip,  not  100  percent  as  stated  by  dod.  dod  may  have  confused  the 
AN/SLQ-32  with  the  AN/ALQ-135  Quick  Reaction  Capability  Jammer,  of 
which  100  percent  of  the  total  program  quantity  was  procured  during 
low-rate  initial  production  (Lrap).  Also,  on  the  Roneer  system,  the  Navy 
made  a  major  commitment  to  procmrement — ^whether  it  was  under  qjecial 
circumstances  or  lrip — ^without  any  assurance  of  the  system’s  adequacy. 
Nevertheless,  in  our  view,  the  cited  systems  illustrate  one  of  the  more 
severe  adverse  impacts  of  beginning  lrip  prematurely,  that  is,  the 
unplanned  termination  of  a  program  after  procurement  of  a  few  costly  but 
imsatisfactory  systems.  Our  proposed  solution  is  to  obtain  at  least  some 
confidence  in  the  system’s  adequacy  in  an  operational  environment  before 
any  commitment  is  made  to  production. 

3.  We  have  clarified  the  description  of  our  review  methodology  and  have 
clearly  pointed  out  that  the  examples  used  in  the  report  are  only  a 
sampling  of  what  we  consider  to  be  the  widespread  problem  of  premature 
commitment  to  production.  Om:  conclusions  and  recommendations  are 
based  on  our  large  body  of  weapon  system  reviews  over  the  past  decade 
or  so  plus  additional  analyses  of  the  current  selected  acquisition  report 
systems  and  recent  Defense  Acquisition  Board  decisions.  'The  dod 
Inspector  General  (dod-ig)  has  arrived  at  similar  conclusions  based  on  its 
independent  reviews.  'The  problems  discussed  in  this  report  are  not  new 
and  continue  to  regularly  occur  in  small  and  large  programs  and  programs 
from  aU  the  services.  The  recommendations  are  consistent  with  our 
previous  recommendations  as  well  as  those  made  by  several  “blue  ribbon” 
panels  on  defense  acquisition  matters,  including  the  Packard  Commission. 
While  dod’s  reluctance  to  adequately  address  the  problem  is  not  new,  we 
are  very  concerned  that  dod  now  appears  to  question  the  significance  of 
the  problem. 

4.  'The  statements  in  our  report  that  (a)  dod  acquisition  policies  permit  lrip 
to  start  without  doing  any  operational,  test  and  evaluation  (ot&e)  and  (b) 
there  are  no  specific  principles  or  guidelines  on  when  or  how  programs 
should  begin  lrip  are  accurate  and  are  not  sweeping  generalizations.  We 
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agree  that  the  regulations  do  mention  lrip  in  many  places;  however,  the 
guidance  is  neither  clear  nor  specific,  and  implementation  has  not  been 
effective  in  controlling  the  commitment  to  production. 

5.  The  current  legislation  contains  no  specific  controls  on  the  start  or 
continuation  of  trap;  10  U.S.C.  2400  defines  the  purposes  of  lrip  but  does 
not  specify  when  and  under  what  circumstances  it  may  begin  or  continue. 

6.  Our  position  would  not  cause  any  production  stretchouts  or  production 
stoppages  and,  for  systems  that  can  readily  demonstrate  their  key 
performance  parameters,  only  a  brief  delay  in  starting  production.  In  fact, 
production  stretchouts  and/or  stoppages  are  usually  the  result  of  problems 
identified  during  operational  testing  performed  after,  rather  than  before, 
the  start  of  production.  We  believe  that  it  is  reasonable  to  delay  the  start  of 
production  to  obtain  assurance  that  the  system  performs  as  intended, 
thereby  avoiding  the  potential  for  stretchouts  or  delays  when  subsequent 
operational  testing  identifies  system  shortcomings. 

7.  Although  DOD  may  want  to  deemphasize  realistic  operational  testing 
during  the  acquisition  of  defense  systems,  it  has  provided  no  factual  basis 
to  state  that  test  and  evaluation  is  a  cause  or  even  a  contributing  factor  to 
the  continuing  problems  in  the  acquisition  of  defense  systems. 

Streamlining  of  the  acquisition  process  should  not  be  used  as  an  excuse 
for  buying  defective  weapon  systems. 

8.  Enclosures  2  and  3  are  not  reproduced  here  because  they  are  lengthy 
and  address  many  issues  not  addressed  in  this  report. 

9.  Developmental  test  agencies  are  elements  of  the  dod  acquisition 
community  and,  in  that  sense,  are  not  independent.  In  fact,  their  lack  of 
independence  was  a  key  factor  in  creating  the  office  of  the  Director, 
Operational  Test  and  Evaluation  (dot&e).  While  the  independence  of 
testers  is  a  key  point,  we  believe  that  early  testing  in  a  realistic  operational 
environment  is  crucial  to  the  identification  of  weaknesses  in  the  system. 
Developmental  testing  is  important  in  itself,  but  it  does  not  expose  the 
system  to  the  rigors  of  the  operational  environment. 

10.  Operational  assessments  are  not  typically  based  on  the  results  of 
realistic  field  testing  and,  therefore,  do  not  provide  the  quantity  and  quality 
of  input  to  decisionmakers  that  we  believe  is  necessary. 
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11.  We  believe  the  cited  dod  policy  is  flawed  in  that  a  system  could  be 
approved  to  begin  lrip  and  yet  not  be  ^proved  as  ready  for  operational 
testing,  hi  our  view,  if  a  system  is  not  ready  to  do  well  in  operational 
testing,  it  is  not  ready  to  begin  lrip.  Our  recommendation  would  require 
operational  testing  prior  to  lrip,  which  could  add  some  cost  and  time  up  to 
that  point.  However,  we  believe  that  additional  investment  would  be  well 
spent  in  terms  of  improved  performance  and  reliability  when  the  system  is 
deployed  and  could  help  avoid  the  costly  modifications  and  retrofits  that 
often  result  firom  dod  committing  to  production  of  systems  without  any 
knowledge  about  their  operational  performance. 

12.  It  is  important  to  have  a  milestone  review  at  the  lrip  decision  point 
because  the  decisionmakers  have  more  flexibility  to  change  the  program 
at  that  point.  As  we  state  in  this  report,  the  decision  to  start  lrip,  in  many 
cases,  is  also  the  de-facto  fuU-rate  production  decision. 

13.  We  recognize  that  our  recommendations  are  at  odds  with  current 
acquisition  practices  for  many  programs.  However,  dod  should  recognize 
that,  in  today’s  environment,  highly  concurrent  acquisition  strategies  are 
no  longer  necessary  and  that  it  is  technically  feasible  for  prototypes  to  be 
sufficiently  mature  to  be  operationally  tested  and  to  meet  their  key 
parameters  before  lrip.  We  believe  that,  in  most  cases,  production 
representative  articles  can  be  assembled  and  realistic  operational  testing 
conducted  before  lrip  begins.  If  a  system  is  not  ready  for  operational 
testing,  it  should  not  be  ready  for  production. 

14.  TWs  recommendation  is  aimed  at  those  very  large  programs  for  which 
it  may  not  be  feasible  or  cost-effective  to  produce  prototypes  without 
starting  lrip.  While  we  recognize  the  potential  for  integration  and  interface 
problems  among  subsystems,  we  question  the  need  to  begin  production  of 
the  primary  system  before  sufficient  confidence  is  available  that  the 
subsystems  perform  adequately  and  are  reliable. 

16.  In  response  to  the  dod-ig  report,  dod  conceded  that  more  attention  may 
be  needed  to  the  lrip  decision  and  that  some  sort  of  controls  may  be 
needed  over  the  start  and  continuation  of  lrip.  We  believe  that,  to  be 
effective,  such  controls  need  to  be  specific  rather  than  a  generic  call  for 
more  emphasis. 

16.  Our  positions  would  cause  the  stetchout  of  only  those  production 
programs  in  which  the  system  was  not  able  to  complete  its  required 
operational  testing.  In  numerous  cases,  lrip  was  continued  indefinitely 
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because  of  the  inability  to  complete  operational  testing,  resulting  in  costly 
retrofits  of  produced  units  and/or  the  fielding  of  unsatisfactory  systems. 
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Military  Airlift:  The  C-17  Program  Update  and  Proposed  Settlement 
(gao/t-nsiad-94-166,  Apr.  19, 1994). 

Acquisition  Reform:  Role  of  Test  and  Evaluation  in  System  Acquisition 
Should  Not  Be  Weakened  (gao/t-nsiad-94-124,  Mar.  22, 1994). 

Army  Acquisition:  Information  on  the  Status  and  Performance  of  the 
Javelin  Antitank  Weapon  (gao/nsiad-94-122br.  Mar.  9, 1994). 


Battlefield  Automation:  Premature  Acquisition  of  the  Army’s  Combat 
Service  Support  Control  System  (gao/nsiad-94-51,  Feb.  4, 1994). 

Navy  Aviation:  V-22  Development — Schedule  Extended,  Performance 
Reduced,  and  Costs  Increased  (gao/nsiad-94-44,  Jan.  13, 1994). 

Strategic  Bomber  Issues  Related  to  the  B-lB’s  Availability  and  Ability  to 
Perform  Conventional  Missions  (gao/nsiad-94-81,  Jan.  10, 1994). 

Unmanned  Aerial  Vehicles:  Performance  of  Short-Range  System  Still  in 
Question  (gao/nsiad-94^6,  Dec.  15, 1993). 

Army  Acquisition:  Problems  With  the  Sense  and  Destroy  Armor  Munition 
(GAO/NSIAD-94-59,  Nov.  23, 1993). 

Missile  Development:  TSSAM  Production  Should  Not  Be  Started  as 
Planned  (gao/nsiad-94-52,  Oct.  8, 1993). 

Test  and  Evaluation:  pod  Has  Been  Slow  in  Improving  Testing  of 
Software-Intensive  Systems  (gao/nsiad-93-1989;  Sept.  29, 1993). 


Electronic  Warfare:  Inadequate  Testing  Led  to  Faulty  SLQ-32s  on  Ships 
(GAO/NSIAD-93-272,  Aug.  19,  1993). 

Army  Acquisition:  Medium  Truck  Program  Is  Not  Practical  and  Needs 
Reassessment  (gao/nsiad-93-232,  Aug.  5, 1993). 

Army  Acquisition:  More  Testing  Needed  to  Solve  Heavy  Equipment 
Transporter  System  Problems  (gao/nsiad-93-228,  July  16, 1993). 

Battlefield  Automation:  More  Testing  and  Analysis  Needed  Before 
Production  of  Air  Defense  Radar  (gao/nsiad-93-175,  July  30, 1993). 
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Tactical  Intelligence:  Joint  STARS  Needs  Current  Cost  and  Operational 
Effectiveness  Analysis  (gao/nsiad-93-ii7,  Apr.  28, 1993). 


Electronic  Warfare:  Laser  Warning  System  Production  Should  Be  Limited 
(gao/nsiad-93-14,  Jan.  25,  1993). 

Weapons  Acquisition:  A  Rare  Opportunity  for  Lasting  Change 
(gao/nsiad-93-15,  Dec.  1992). 

Military  Communications:  Joint  Tactical  Information  Distribution  System 
Issues  (gao/nsiad-93-16,  Nov.  12,  1992). 

Battlefield  Automation:  Planned  Production  Decision  for  Army  Control 
System  Is  Premature  (gao/nsiad-92-151,  Aug.  10,  1992). 

Strategic  Bombers:  Need  to  Redefine  Requirements  for  B-IB  Defensive 
Avionics  System  (gao/nsiad-92-272,  July  17, 1992). 

Comanche  Helicopter:  Program  Needs  Reassessment  Due  to  Increased 
Unit  Cost  and  Other  Factors  (gao/nsiad-92-204.  May  27, 1992). 

Embedded  Computer  Systems:  Software  Development  Problems  Delay  the 
Army’s  Fire  Direction  Data  Manager  (gao/imtec-92-32.  May  11,  1992). 


Embedded  Computer  Systems:  Significant  Software  Problems  on  C-17 
Must  Be  Addressed  (gao/imtec-92-48.  May  7,  1992). 

Strategic  Missiles:  ACM  Program,  Opportunity  for  Additional  Savings 
(gao/nsiad-92-154.  May  6, 1992). 

Operation  Desert  Storm:  Apache  HeUcopter  Was  Considered  Effective  in 
Combat,  but  Rehability  Problems  Persist  (gao/nsiad-92-146,  Apr.  20,  1992). 

Electronic  Warfare:  Faulty  Test  Equipment  Impairs  Readiness  of  Army 
Hehcopters  (gao/nsiad-92-128,  Apr.  17,  1992). 

Embedded  Computer  Systems:  F-14D  Aircraft  Software  Is  Not  Reliable 
(gaoamtec-92-21,  Apr.  2, 1992). 

Strategic  Defense  Initiative:  Estimates  of  Brilhant  Pebbles’  Effectiveness 
Are  Based  on  Many  Unproven  Assumptions  (gao/nsla.d-92-91.  Mar.  27, 1992). 
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Electronic  Warfare:  Established  Criteria  Not  Met  for  Airborne 
Self-Protection  Jammer  Production  (gao/nsiad-92-103,  Mar.  23, 1992). 

Army’s  M109  Howitzer:  Required  Testing  Should  Be  Completed  Before 
FuU-Rate  Production  (nsiad-92-44,  Jan.  23, 1992). 

Longbow  Apache  Helicopter  Key  Factors  Used  to  Measure  Progress  in 
Development  Need  to  Be  Changed  (nsiad-92-43,  Nov.  21, 1991). 

Submarine  Combat  System:  BSY-2  Development  Risks  Must  Be  Addressed 
and  Production  Schedule  Reassessed  (gao/imtec-91-30,  Aug.  22, 1991). 

Munitions  Procurement:  Resolve  Questions  Before  Proceeding  With 
Sensor  Fuzed  Weapon  Production  (gao/nsiad-91-235,  Aug.  16, 1991). 

Battlefield  Automation:  Army  Tactical  Command  and  Control  System 
Development  Problems  (gao/nsiad-91-172,  July  31, 1991). 


Defense  Management:  Stronger  Oversight  of  Joint  Service  Imagery 
Processing  System  Needed  (gao/nsiad-91-164,  July  26, 1991). 

The  B-2  Program:  Procurement  Decisions  Should  Be  Based  on 
Demonstrated  Performance  (gao/t-nsiad-9145,  July  17, 1991). 

Missile  Procurement:  AMEAAM’s  Reliability  Is  Improving,  but  Production 
Challenges  Remain  (gao/nsiad-91-209,  June  20, 1991). 

Battlefield  Automation:  Army  Needs  to  Reevaluate  Air  Defense  Radar 
Acquisition  I^rograms  (gao/nsiad-91-91.  May  3, 1991). 


Global  Positioning  System:  Production  Should  Be  Limited  Until  Receiver 
Rehability  Problems  Are  Resolved  (gao/nsiad-91-74.  Mar.  20, 1991). 

Electronic  Warfare:  Early  Production  of  Tacit  Rainbow  Missile  Not 
Warranted  (gao/nsiad-91-71.  Mar.  8, 1991). 

Letter  Report  to  Senzitor  Roth  on  F-117A  Acquisition  Management 
Practices  (B-238891,  Jan.  11,  1991). 

Antisubmarine  Warfare:  Tactical  Surveillance  Sonobuoy  and  Related 
Software  Need  to  Be  Tested  Together  (gao/nsiad-91-41,  Jan.  9, 1991). 
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T-45  Training  System:  Navy  Should  Reduce  Risks  Before  Procuring  More 
Aircraft  (gao/nsiad-91-46,  Dec.  14,  1990). 

Army  Acquisition:  Air  Defense  Antitank  System  Did  Not  Meet  Operational 
Test  Objectives  (gao/nsiad-91-51,  Dec.  10, 1990). 


(707065) 
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